Air combat maneuvering method based on parallel self-play

ABSTRACT

The present disclosure provides an air combat maneuvering method based on parallel self-play, including the steps of constructing a UAV (unmanned aerial vehicle) maneuver model, constructing a red-and-blue motion situation acquiring model to describe a relative combat situation of red and blue sides, constructing state spaces and action spaces of both red and blue sides and a reward function according to a Markov process, followed by constructing a maneuvering decision-making model structure based on a soft actor-critic (SAC) algorithm, training the SAC algorithm by performing air combat confrontations to realize parallel self-play, and finally testing a trained network, displaying combat trajectories and calculating a combat success rate. The level of confrontations can be effectively enhanced and the combat success rate of the decision-making model can be increased.

This application claims the benefit of Chinese Patent Application No. 202110364109.3, filed on Apr. 3, 2021, which is hereby incorporated by reference in its entirety.

FIELD

The present disclosure pertains to the technical field of unmanned aerial vehicles (UAVs) and particularly relates to an air combat maneuvering method.

BACKGROUND

Autonomous air combat maneuvering decision-making is a process of simulating air combat decisions of pilots in various air combat situations and automatically generating maneuvering decisions for aerial vehicles (including manned and unmanned) based on mathematical optimization, artificial intelligent algorithms and the like.

According to different methods, there may be two different types of methods, traditional and intelligent, for common maneuvering decision-making of UAVs. Traditional methods refer to the process of optimal decision-making by using expert knowledge, formula derivation, influence diagrams, etc. Such methods are more concentrated on priori knowledge or mathematical operation, and often lack a self-optimization process for decision-making. Intelligent methods refer to those methods capable of self-learning and self-optimization by using approaches such as genetic algorithms, Bayes and artificial intelligence, which are used to realize maneuver control of UAVs. Such methods can usually achieve strategy optimization according to situation targets autonomously.

However, when solving the confrontation problem of both parties, these methods have the following disadvantage: since training is performed in a single deterministic environment, policy models obtained through learning by agents would be weak in robustness for being highly adaptive to the current environment and situation. When applied to a new environment or being changed in initial situation, these policy models cannot select rational decision actions accurately. In addition, learning from scratch in a new environment would take a lot of time for training.

SUMMARY

To overcome the shortcomings of the prior art, the present disclosure provides an air combat maneuvering method based on parallel self-play, including the steps of constructing a UAV maneuver model, constructing a red-and-blue motion situation acquiring model to describe a relative combat situation of red and blue sides, constructing state spaces and action spaces of both red and blue sides and a reward function according to a Markov process, followed by constructing a maneuvering decision-making model structure based on a soft actor-critic (SAC) algorithm, training the SAC algorithm by performing air combat confrontations to realize parallel self-play, and finally testing a trained network, displaying combat trajectories and calculating a combat success rate. According to the present disclosure, the level of confrontations can be effectively enhanced and the combat success rate of the decision-making model can be increased.

The technical solution for solving the technical problems in the present disclosure includes the following steps:

step S1: constructing a UAV maneuver model;

step S2: defining our UAV as red side and enemy UAV as blue side; initializing both red and blue sides, and constructing a red-blue motion situation acquiring model to describe a relative combat situation between the red and blue sides;

step S3: constructing state spaces S_(r),S_(b) of both red and blue sides, action spaces A_(r),A_(b) of both red and blue sides and a reward function R according to a Markov process;

step S4: constructing a maneuvering decision-making model structure based on a soft actor-critic (SAC) algorithm;

step S5: initializing a plurality of groups of UAVs on both sides, defining experimental parameters, and training the SAC algorithm by allowing the plurality of groups of UAVs on both sides to perform air combat confrontations using the same maneuvering decision-making model and a same replay buffer to realize parallel self-play; and

step S6: randomly initializing both sides to test a trained network, and displaying combat trajectories; randomly initializing the plurality of groups of UAVs on both sides to test the trained network, and calculating a combat success rate.

Further, the constructing a UAV maneuver model may specifically include the following steps:

supposing an OXYZ coordinate system to be a three-dimensional spatial coordinate system for UAVs, where origin O represents the center of a combat area for UAVs, with X axis pointing to the north, Z axis pointing to the east and Y axis pointing in a vertical upward direction;

regarding a UAV as a mass point and establishing equations of motion for the UAV as follows:

$\left\{ \begin{matrix} X_{t + {dT}} & = & {X_{t} + {V_{t + {dT}}*{\cos\left( \theta_{t + {dT}} \right)}*{\cos\left( \varphi_{t + {dT}} \right)}*{dT}}} \\ Y_{t + {dT}} & = & {Y_{t} + {V_{t + {dT}}*{\sin\left( \theta_{t + {dT}} \right)}*{dT}}} \\ Z_{t + {dT}} & = & {Z_{t} + {V_{t + {dT}}*{{\cos\left( \theta_{t + {dT}} \right)}**{\sin\left( \varphi_{t + {dT}} \right)}}{dT}}} \end{matrix} \right.$ $\left\{ \begin{matrix} {V_{t + {dT}} = {V_{t} + {{dv}*{dT}}}} \\ {\theta_{t + {dT}} = {\theta_{t} + {d\theta*{dT}}}} \\ {\varphi_{t + {dT}} = {\varphi_{t} + {d\varphi*{dT}}}} \end{matrix} \right.$

where t denotes current time; dT denotes an integration step size of the UAV; [X_(t),Y_(t),Z_(t)], [X_(t+dT),Y_(t+dT),Z_(t+dT)] denote coordinate position components of the UAV at time t and time t+dT, respectively; V_(t), V_(t+dT) denote velocities of the UAV at time t and time t+dT, respectively; pitch angles θ_(t), θ_(t+dT) are included angles between velocity vectors of the UAV at time t and time t+dT, and XOZ plane; heading angles φ_(t), φ_(t+dT) are included angles between projection vectors of the velocity vectors of the UAV at time t and time t+dT on the XOZ plane, and the positive X axis; dυ denotes an acceleration of the UAV; dθ denotes a pitch angle variation of the UAV; and dθ denotes a heading angle variation of the UAV;

Further, the step S2 may specifically include the following steps:

describing the relative situation of both sides acquired by the red-and-blue motion situation acquiring model with {right arrow over (D)}, d and q, where {right arrow over (D)} denotes a position vector between the red side and the blue side in a direction from the red side to the blue side; d denotes a distance between the red side and the blue side; q denotes a relative azimuth angle, namely an included angle between the velocity vector {right arrow over (V)}_(r) and the distance vector {right arrow over (D)} of the red side; and

denoting the combat situation of the blue side relative to the red side by {right arrow over (D)}_(r), d and q_(r) and the combat situation of the red side relative to the blue side by {right arrow over (D)}_(b), d and q_(b), where {right arrow over (D)}_(r) denotes a position vector between the red side and the blue side in a direction from the red side to the blue side; {right arrow over (D)}_(b) denotes a position vector between the blue side and the red side in a direction from the blue side to the red side; q_(r) denotes a relative azimuth angle of the blue side to the red side; and q_(b) denotes a relative azimuth angle of the red side to the blue side; and {right arrow over (D)}_(r), {right arrow over (D)}_(b), d, q_(r) and q_(b) are calculated as follows:

${\overset{\rightarrow}{D}}_{r} = \left( {{X_{b} - X_{r}},{Y_{b} - Y_{r}},{Z_{b} - Z_{r}}} \right)$ ${\overset{\rightarrow}{D}}_{b} = \left( {{X_{r} - X_{b}},{Y_{r} - Y_{b}},{Z_{r} - Z_{b}}} \right)$ $d = \sqrt{\left( {X_{b} - X_{r}} \right)^{2} + \left( {Y_{b^{-}}Y_{r}} \right)^{2} + \left( {Z_{b} - Z_{r}} \right)^{2}}$ $q_{r} = {{a{\cos\left( \frac{{\overset{\rightarrow}{D}}_{r} \times \overset{\rightarrow}{V_{r}}}{{\overset{\rightarrow}{D}}{\overset{\rightarrow}{V_{r}}}} \right)}} = {a{\cos\left( \frac{{\left( {X_{b} - X_{r}} \right)v_{xr}} + {\left( {Y_{b} - Y_{r}} \right)v_{yr}} + {\left( {Z_{b} - Z_{r}} \right)v_{zr}}}{d^{*}v_{r}} \right)}}}$ $q_{b} = {{{acos}\left( \frac{{\overset{\rightarrow}{D}}_{b} \times \overset{\rightarrow}{V_{b}}}{{\overset{\rightarrow}{D}}{\overset{\rightarrow}{V_{b}}}} \right)} = {a{\cos\left( \frac{{\left( {X_{r} - X_{b}} \right)v_{xb}} + {\left( {Y_{r} - Y_{b}} \right)v_{yb}} + {\left( {Z_{r} - Z_{b}} \right)v_{zb}}}{d^{*}v_{b}} \right)}}}$

where {right arrow over (R)}_(r)=(X_(r),Y_(r),Z_(r)), {right arrow over (V)}_(r)=(υ_(xr),υ_(yr),υ_(zr)), υ_(r), θ_(r) and φ_(r) are the position vector, velocity vector, velocity, pitch angle and heading angle of the red side, respectively; and {right arrow over (R)}_(b)=(X_(b),Y_(b),Z_(b)), {right arrow over (V)}_(b)=(υ_(xb),υ_(yb),υ_(zb)), υ_(b), θ_(b) and φ_(b) are the position vector, velocity vector, velocity, pitch angle and heading angle of the blue side, respectively.

Further, the step S3 may specifically include the following steps:

defining the state space of the red UAV as S_(r)=[X_(r),Y_(r),Z_(r),υ_(r),θ_(r),φ_(r),d,q_(r)] and the state space of the blue UAV as S_(b)=[X_(b),Y_(b),Z_(b),υ_(b),θ_(b),φ_(b),d,q_(b)];

defining the action space of the red UAV as A_(r)=[dυ_(r),dφ_(r),dθ_(r)]and the action space of the blue UAV as A_(b)=[dυ_(b),dφ_(b),dθ_(b)]; and

forming the reward function R with a distance reward function R_(d) and an angle reward function R_(q), R=w₁*R_(d)+w₂*R_(a), where w₁,w₂ denote weights of a distance reward and an angle reward;

the distance reward function R_(d) is expressed as:

R_(d1) = −d/(5 * D_(max)) $R_{d2} = \left\{ \begin{matrix} {3,} & {{{if}D_{\min}} < d < D_{\max}} \\ {{- 1},} & {{{if}d} < D_{\min}} \end{matrix} \right.$ R_(d) = R_(d1) + R_(d2)

where R_(d1) denotes a continuous distance reward, while R_(d2) denotes a sparse distance reward; and D_(min) denotes a minimum attack range of a missile carried by the red side, while D_(max) denotes a maximum attack range of the missile carried by the red side; and

the angle reward function R_(q) is expressed as:

R _(q1) =−q/180

R _(q2)=3, if q<q _(max)

R _(q) =R _(q1) +R _(q2)

where R_(q1) denotes a continuous angle reward, while R_(q2) denotes a sparse angle reward; and q_(max) denotes a maximum off-boresight launch angle of the missile carried by the red side.

Further, the constructing a maneuvering decision-making model structure based on a SAC algorithm may specifically include the following steps:

generating maneuver control quantities for both red and blue sides by the maneuvering decision-making model based on the SAC algorithm using a SAC method, to allow the red and blue sides to maneuver; and

implementing the SAC algorithm by neural networks including an replay buffer M, one Actor neural network π_(θ), two Soft-Q neural networks Q₁₀₀ ₁ and Q_(φ2), two Target Soft-Q networks Q_(100 ′) ₁ and Q_(φ′) ₂ , where θ, φ₁, φ₂, φ′₁, φ′₂ denote weights of the respective networks;

where the Actor neural network π_(θ) receives an input of a state value s_(t) ^(r) of the red side or a state value s_(t) ^(b) of the blue side and generates outputs of mean μ(μ_(r),μ_(b)) and variance σ(σ_(r),σ_(b)); noise τ is generated by sampling from a standard normal distribution; an action a_(t) ^(r) of the red side or an action a_(t) ^(b) of the blue side is generated from the mean μ, variance σ and noise τ; the action a_(t) ^(r) or a_(t) ^(b) is limited to a range of(−1,1) by using a tanh function, and the process of generating the action is shown below:

μ_(r),σ_(r)=π_(θ)(s_(t) ^(r))

μ_(b),σ_(b)=π_(θ)(s_(t) ^(b))

a _(t) ^(r) =N(μ_(r),σ_(r) ²)=μ_(r)+σ_(r)*τ

a _(t) ^(b) =N(μ_(b),σ_(b) ²)=μ_(b)+σ_(b)*τ

a_(t) ^(r)=tanh(a_(t) ^(r))

a_(t) ^(b)=tanh(a_(t) ^(b))

the Soft-Q neural networks Q_(θ1) and Q_(θ2) receive inputs of a state value and an action value and output Q values predicted by the neural networks; the Target Soft-Q neural networks Q_(φ′) ₁ and Q_(φ′) ₂ have a same structure with and are different in network weight from the Soft-Q neural networks; the Soft-Q neural networks are configured to generate predicted Q values and select an action, while the Target Soft-Q neural networks are configured to generate target Q values and evaluate the action; the weights of the Soft-Q neural networks are updated in real time and copied to the Target Soft-Q neural networks after multiple iterations; and

each of the Actor, Soft-Q and Target Soft-Q networks is a fully-connected neutral network having l hidden layers, with n neurons in each hidden layer and an activation function ReLU.

Further, the step S5 may specifically include the following steps:

when initializing a plurality of groups of UAVs on both sides, with initial positions within the combat area, and setting an initial velocity range, an initial pitch angle range and an initial heading angle range; and

the steps of training the SAC algorithm by performing air combat confrontations to realize parallel self-play are as follows:

step S51: defining the number env_num of parallel self-play environments, defining the number batch_size of batch training sample groups, defining a maximum simulation step size N, initializing step=1, initializing env=1, initializing initial situations of both sides, and obtaining an initial state s_(t) ^(r) of the red side and an initial state s_(t) ^(b) of the blue side;

step S52: randomly generating Actor network weight θ, Soft-Q network weights φ₁, φ₂ initializing the policy network π_(θ) and the two Soft-Q networks Q_(φ1), Q_(φ2), supposing φ′₁=φ₁, φ′₂=φ₂, and initializing the Target Soft-Q networks Q_(100 ′) ₁ , Q_(φ′) ₂ with φ′₁ ,φ′₂ as network weights;

step S53: inputting a state s_(t) ^(r) of the red side to the Actor network to output a mean μ₄ and a variance σ_(r), obtaining an action a_(t) ^(r) that fits the action space A_(r) in step S3 from the process of generating the action in step S4, obtaining a new state by the red side after performing the action s_(t+1) ^(r), and obtaining a reward value r_(t) ^(r) according to the reward function R in step S3; inputting a state s_(t) ^(b) of the blue side to the Actor network to output a mean μ_(b) and a variance σ_(b), obtaining an action a_(t) ^(b) that fits the action space A_(b) in step S3 from the process of generating the action in step S4, obtaining a new state by the blue side after performing the action s_(t+1) ^(b), and obtaining a reward value r_(t) ^(b) according to the reward function R in step S3; and storing tuple <s_(t) ^(r),a_(t) ^(r),s_(t+1) ^(r),r_(t) ^(r)> and tuple <s_(t) ^(b),a_(t) ^(b),s_(t+1) ^(b),r_(t) ^(b)> in the replay buffer M;

step S54: determining whether env is greater than env_num, and if yes, proceeding to step S55; otherwise, incrementing env by 1, and skipping to step S51;

step S55: when the number of experience groups in the replay buffer is greater than batch_size, randomly sampling batch_size groups of experience to update parameters of the Actor and Soft-Q neutral networks in the SAC algorithm, and updating a regularization coefficient α;

step S56: determining whether step is greater than N, and if yes, proceeding to step S57; otherwise, incrementing step by 1, s_(t) ^(r)==s_(t+1) ^(r), s_(t) ^(b)=s_(t+1) ^(b), and skipping to step S53; and

step S57: determining whether the algorithm converges or whether training episodes are met, and if yes, ending the training and obtaining the trained SAC algorithm model; otherwise, skipping to step S51.

Further, the step S6 may specifically include the following steps:

step S61: initializing the initial situations of both sides, and obtaining the initial states s_(t) ^(r), s_(t) ^(b) of the red and blue sides;

step S62: separately recording the states s_(t) ^(r), s_(t) ^(b), inputting the states s_(t) ^(r), s_(t) ^(b) to the Actor neutral network of the trained SAC algorithm model to output actions a_(t) ^(r), a_(t) ^(b) of the red and blue sides, and obtaining new states s_(t+1) ^(r), s_(t+1) ^(b) after performing the actions by both sides;

step S63: determining whether either of both sides succeeds in engaging in combat, and if yes, ending; otherwise, supposing s_(t) ^(r)=s_(t+1) ^(r) and s_(t) ^(b)=s_(t+1) ^(b), and skipping to step S62;

step S64: plotting combat trajectories of both sides according to the recorded states s_(t) ^(r), s_(t) ^(b);

step S65: initializing the initial situations of n groups of UAVs on both sides, performing steps S62 to S63 on each group of UAVs on both sides, and finally recording whether either of both sides succeeds in engaging in combat, with the number of times of successfully engaging in combat being denoted as num; and

step S66: calculating num/n, namely a final combat success rate, to indicate the generalization capability of the decision-making model.

Further, in the step S5, the initial velocity range may be set as [50 m/s, 400 m/s], and the initial pitch angle range as [−90 °,90°] and the initial heading angle range as [−180°,180°].

The present disclosure has the following beneficial effects:

1. According to the present disclosure, a plurality of battlefield environments are introduced during self-play, and samples and strategies can be shared among the battlefield environments. Thus, maneuvering strategies can be overall optimized.

2. The parallel self-play algorithm proposed in the present disclosure can effectively enhance the level of confrontations and increase the combat success rate of the decision-making model.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flowchart of a method according to the present disclosure.

FIG. 2 is a schematic diagram of self-play in a method according to the present disclosure.

FIG. 3 is a chart of combat trajectories according to an embodiment of the present disclosure.

FIG. 4 is a combat success rate curve according to an embodiment of the present disclosure.

DETAILED DESCRIPTION

The present disclosure is further described below in conjunction with the accompanying drawings and embodiments.

As shown in FIG. 1, an air combat maneuvering method based on parallel self-play includes the following steps:

step S1: constructing a UAV maneuver model;

step S2: defining our UAV as red side and enemy UAV as blue side; initializing both red and blue UAVs, and constructing a red-and-blue motion situation acquiring model to describe a relative combat situation of the red and blue sides;

step S3: constructing state spaces S_(r),S_(b) of both red and blue sides, action spaces A_(r),A_(b) of both red and blue sides and a reward function R according to a Markov process;

step S4: constructing a maneuvering decision-making model structure based on a SAC algorithm;

step S5: initializing a plurality of groups of UAVs on both sides, defining experimental parameters, and training the SAC algorithm by allowing the plurality of groups of UAVs on both sides to perform air combat confrontations using the same maneuvering decision-making model and a same replay buffer to realize parallel self-play; and

step S6: randomly initializing both sides to test a trained network, and displaying combat trajectories; randomly initializing a plurality of groups of UAVs on both sides to test the trained network, and calculate a combat success rate.

Further, the constructing a UAV maneuver model includes the following specific steps:

The position information of UAVs of both sides is updated according to equations of motion for UAVs, so that maneuvering can be realized. Furthermore, the information of both sides is provided to the both-side situation acquiring model to calculate corresponding situations.

An OXYZ coordinate system is supposed to be a three-dimensional spatial coordinate system for UAVs, where origin O represents the center of a combat area for UAVs, with X axis pointing to the north, Z axis pointing to the east and Y axis pointing in a vertical upward direction.

A UAV is regarded as a mass point and equations of motion for the UAV are established as follows:

$\left\{ \begin{matrix} X_{t + {dT}} & = & {X_{t} + {V_{t + {dT}}*{\cos\left( \theta_{t + {dT}} \right)}*{\cos\left( \varphi_{t + {dT}} \right)}*{dT}}} \\ Y_{t + {dT}} & = & {Y_{t} + {V_{t + {dT}}*{\sin\left( \theta_{t + {dT}} \right)}*{dT}}} \\ Z_{t + {dT}} & = & {Z_{t} + {V_{t + {dT}}*{{\cos\left( \theta_{t + {dT}} \right)}**{\sin\left( \varphi_{t + {dT}} \right)}}{dT}}} \end{matrix} \right.$ $\left\{ \begin{matrix} {V_{t + {dT}} = {V_{t} + {{dv}*{dT}}}} \\ {\theta_{t + {dT}} = {\theta_{t} + {d\theta*{dT}}}} \\ {\varphi_{t + {dT}} = {\varphi_{t} + {d\varphi*{dT}}}} \end{matrix} \right.$

where t denotes current time; dT denotes an integration step size of the UAV; [X_(t),Y_(t),Z_(t)], [X_(t+dT),Y_(t+dT),Z_(t+dT)] denote coordinate position components of the UAV at time t and time t+dT, respectively; V_(t), V_(t+dT) denote velocities of the UAV at time t and time t+dT, respectively; pitch angles θ_(t), θ_(t+dT) are included angles between velocity vectors of the UAV at time t and time t+dT, and XOZ plane; heading angles φ_(t), φ_(t+dT) are included angles between projection vectors of the velocity vectors of the UAV at time t and time t+dT on the XOZ plane, and the positive X axis; dυ denotes an acceleration of the UAV; dθ denotes a pitch angle variation of the UAV; and dφ denotes a heading angle variation of the UAV;

Further, the step S2 includes the following specific steps:

The red-and-blue motion situation acquiring model can calculate a relative situation according to red and blue state information and provide the relative situation to a maneuvering decision-making module based on a deep reinforcement learning method for decision-making.

The relative situation of both sides acquired by the red-and-blue motion situation acquiring model is described with {right arrow over (D)}, d and q, where {right arrow over (D)} denotes a position vector between the red side and the blue side in a direction from the red side to the blue side; d denotes a distance between the red side and the blue side; q denotes a relative azimuth angle, namely an included angle between the velocity vector {right arrow over (V)}_(r) and the distance vector {right arrow over (D)} of the red side.

The combat situation of the blue side relative to the red side is denoted by {right arrow over (D)}_(r), d and q_(r) and the combat situation of the red side relative to the blue side is denoted by {right arrow over (D)}_(b), d and q_(b), where{right arrow over (D)}_(r) denotes a position vector between the red side and the blue side in a direction from the red side to the blue side; {right arrow over (D)}_(b) denotes a position vector between the blue side and the red side in a direction from the blue side to the red side; q_(r) denotes a relative azimuth angle of the blue side to the red side; and q_(b) denotes a relative azimuth angle of the red side to the blue side.

{right arrow over (D)}_(r), {right arrow over (D)}_(b), d, q_(r) and q_(b) are calculated as follows:

${\overset{\rightarrow}{D}}_{r} = \left( {{X_{b} - X_{r}},{Y_{b} - Y_{r}},{Z_{b} - Z_{r}}} \right)$ ${\overset{\rightarrow}{D}}_{b} = \left( {{X_{r} - X_{b}},{Y_{r} - Y_{b}},{Z_{r} - Z_{b}}} \right)$ $d = \sqrt{\left( {X_{b} - X_{r}} \right)^{2} + \left( {Y_{b^{-}}Y_{r}} \right)^{2} + \left( {Z_{b} - Z_{r}} \right)^{2}}$ $q_{r} = {{a{\cos\left( \frac{{\overset{\rightarrow}{D}}_{r} \times \overset{\rightarrow}{V_{r}}}{{\overset{\rightarrow}{D}}{\overset{\rightarrow}{V_{r}}}} \right)}} = {a{\cos\left( \frac{{\left( {X_{b} - X_{r}} \right)v_{xr}} + {\left( {Y_{b} - Y_{r}} \right)v_{yr}} + {\left( {Z_{b} - Z_{r}} \right)v_{zr}}}{d^{*}v_{r}} \right)}}}$ $q_{b} = {{{acos}\left( \frac{{\overset{\rightarrow}{D}}_{b} \times \overset{\rightarrow}{V_{b}}}{{\overset{\rightarrow}{D}}{\overset{\rightarrow}{V_{b}}}} \right)} = {a{\cos\left( \frac{{\left( {X_{r} - X_{b}} \right)v_{xb}} + {\left( {Y_{r} - Y_{b}} \right)v_{yb}} + {\left( {Z_{r} - Z_{b}} \right)v_{zb}}}{d^{*}v_{b}} \right)}}}$

where {right arrow over (R)}_(r)=(X_(r),Y_(r),Z_(r)), {right arrow over (V)}_(r)=(υ_(xr), υ_(yr), υ_(zr)), υ_(r), θ_(r) and φ_(r) are the position vector, velocity vector, velocity, pitch angle and heading angle of the red side, respectively; and {right arrow over (R)}_(b)=(X_(b),Y_(b),Z_(b)), {right arrow over (V)}_(b)=(υ_(xb), υ_(yb), υ_(zb)), υ_(b), θ_(b) and φ_(b) are the position vector, velocity vector, velocity, pitch angle and heading angle of the blue side, respectively.

Further, the step S3 includes the following specific steps:

The state space of the red UAV is defined as S_(r)=[X_(r),Y_(r),Z_(r),υ_(r),θ_(r),φ_(r),d,q_(r)]and the state space of the blue UAV is defined as S_(b)=[X_(b),Y_(b),Z_(b),υ_(b),θ_(b),φ_(b),d,q_(b)].

The action space of the red UAV is defined as A_(r)=[dυ_(r),dφ_(r),dθ_(r)] and the action space of the blue UAV is defined as A_(b)=[dυ_(b),dφ_(b),dθ_(b)].

The reward function R is formed with a distance reward function R_(d) and an angle reward function R_(q), R=w₁*R_(d)+w₂*R_(a), where w₁,w₂ denote weights of a distance reward and an angle reward.

The distance reward function R_(d) is expressed as:

R_(d1) = −d/(5 * D_(max)) $R_{d2} = \left\{ \begin{matrix} {3,} & {{{if}D_{\min}} < d < D_{\max}} \\ {{- 1},} & {{{if}d} < D_{\min}} \end{matrix} \right.$ R_(d) = R_(d1) + R_(d2)

where R_(d1) denotes a continuous distance reward, while R_(d2) denotes a sparse distance reward; and D_(min) denotes a minimum attack range of a missile carried by the red side, while D_(max) denotes a maximum attack range of the missile carried by the red side.

The angle reward function R_(q) is expressed as:

R _(q1) =−q/180

R _(q2)=3, if q<q _(max)

R _(q) =R _(q1) +R _(q2)

where R_(q1) denotes a continuous angle reward, while R_(q2) denotes a sparse angle reward; and q_(max) denotes a maximum off-boresight launch angle of the missile carried by the red side.

Further, as shown in FIG. 2, the constructing a maneuvering decision-making model structure based on a SAC algorithm includes the following specific steps:

Maneuver control quantities for both red and blue sides are generated by the maneuvering decision-making model based on the SAC algorithm using a SAC method, to allow the red and blue sides to maneuver.

The SAC algorithm is implemented by neural networks including an replay buffer M, one Actor neural network π_(θ), two Soft-Q neural networks Q_(φ1) and Q_(φ2), two Target Soft-Q networks Q_(φ′di 1) and Q_(φ′) ₂ , where θ, φ₁, φ₂, φ′₁, φ′₂ denote weights of the respective networks.

The replay buffer M is an experience replay buffer structure for specially storing experience learned in reinforcement learning.

The Actor neural network π_(θ)receives an input of a state value s_(t) ^(r) of the red side or a state value s_(t) ^(b) of the blue side and generates outputs of mean μ(μ_(r),μ_(b)) and variance σ(σ_(r),σ_(b)). Noise τ is generated by sampling from a standard normal distribution. An action a_(t) ^(r) of the red side or an action a_(t) ^(b) of the blue side is generated from the mean μ, variance σ and noise τ. The action a_(t) ^(r) or a_(t) ^(b) is limited to a range of(−1,1) by using a tanh function, and the process of generating the action is shown below:

μ_(r),σ_(r)=π_(θ)(s_(t) ^(r))

μ_(b),σ_(b)=π_(θ)(s_(t) ^(b))

a _(t) ^(r) =N(μ_(r),σ_(r) ²)=μ_(r)+σ_(r)*τ

a _(t) ^(b) =N(μ_(b),σ_(b) ²)=μ_(b)+σ_(b)*τ

a_(t) ^(r)=tanh(a_(t) ^(r))

a_(t) ^(b)=tanh(a_(t) ^(b))

The Soft-Q neural networks Q_(φ1) and Q_(φ2) receive inputs of a state value and an action value and output Q values predicted by the neural networks. The Target Soft-Q neural networks Q_(φ′) ₁ and Q_(φ′) ₂ have a same structure with and are different in network weight from the Soft-Q neural networks. The Soft-Q neural networks are configured to generate predicted Q values and select an action, while the Target Soft-Q neural networks are configured to generate target Q values and evaluate the action. The weights of the Soft-Q neural networks are updated in real time and copied to the Target Soft-Q neural networks after multiple iterations.

Each of the Actor, Soft-Q and Target Soft-Q networks is a fully-connected neutral network having l hidden layers, with n neurons in each hidden layer and an activation function ReLU.

Further, the step S5 includes the following specific steps:

When initializing a plurality of groups of UAVs on both sides, with initial positions within the combat area, an initial velocity range is set as [50 m/s, 400 m/s], and an initial pitch angle range as [−90°,90°] and an initial heading angle range as [−180°,180°].

The steps of training the SAC algorithm by performing air combat confrontations to realize parallel self-play are as follows:

step S51: defining the number env_num of parallel self-play environments, defining the number batch_size of batch training sample groups, defining a maximum simulation step size N, initialize step=1, initialize env=1, initializing initial situations of both sides, and obtaining an initial state s_(t) ^(r) of the red side and an initial state s_(t) ^(b) of the blue side;

step S52: randomly generating Actor network weight θ, and Soft-Q network weights φ₁, φ₂, initializing the policy network π_(θ) and the two Soft-Q networks Q_(φ1), Q_(φ2), letting φ′₁=φ_(q), φ′₂=φ₂, and initializing the Target Soft-Q networks Q_(φ′) ₁ , Q_(φ′) ₂ with φ′₁,φ′₂ as network weights.

step S53: inputting a state s_(t) ^(r) of the red side to the Actor network to output a mean μ_(r) and a variance σ_(r), obtaining an action a_(t) ^(r) that fits the action space A_(r) in step S3 from the process of generating the action in step S4, obtaining a new state s_(t+1) ^(r) by the red side after performing the action, and obtaining a reward value r_(t) ^(r) according to the reward function R in step S3; inputting a state s_(t) ^(b) of the blue side to the Actor network to output a mean μ_(b) and a variance σ_(b), obtaining an action a_(t) ^(b) that fits the action space A_(b) in step S3 from the process of generating the action in step S4, obtaining a new state s_(t+1) ^(b) by the blue side after performing the action, and obtaining a reward value r_(t) ^(b) according to the reward function R in step S3; and storing tuple <s_(t) ^(r),a_(t) ^(r),s_(t+1) ^(r),r_(t) ^(r)> and tuple <s_(t) ^(b),a_(t) ^(b),s_(t+1) ^(b),r_(t) ^(b)> in the replay buffer M;

step S54: determining whether env is greater than env_num, and if yes, proceeding to step S55; otherwise, incrementing env by 1, and skipping to step S51;

step S55: when the number of experience groups in the replay buffer M is greater than batch_size, randomly sampling batch_size groups of experience to update the parameters of the Actor and Soft-Q neutral networks in the SAC algorithm, and update a regularization coefficient α, where each group of data is redefined as <s_(t),a_(t),s_(t+1),r>. Gradient descent is performed on the loss function of the Actor neutral network and the loss function J_(q)(φ_(i))i=1,2 of the Soft-Q neutral networks with a learning rate lr to update the weights of the Actor neutral network and the Soft-Q neutral networks.

Both Soft -Q functions are defined as minimum output values of the Target Soft-Q networks Q_(φ′) ₁ , Q_(φ′) ₂ , and therefore, the following formula is given:

Q_(φ),(s_(t),a_(t))=min(Q₁₀₀ ₁,(s_(t),a_(t)),Q₁₀₀ ₂,(s_(t),a_(t)))

where Q₁₀₀ ₁,(s_(t),a_(t)),Q₁₀₀ ₂,(s_(t),a_(t)) denote output target Q values of the Target Soft-Q networks Q_(100 ′) ₁ , Q_(100 ′) ₂ , respectively.

The loss function of the Actor neutral network is defined as follows:

J _(π)(θ)=E _(s) _(t) _(˜M, a) _(t) _(˜π) _(θ) [log π_(θ() a _(t) |s _(t))−Q _(φ′)(s _(t) ,a _(t))]

The loss function J_(Q)(φ_(i))i=1,2 of the Soft-Q neutral networks is defined as follows:

${J_{Q}\left( \varphi_{i} \right)} = {E_{{{({s_{t},a_{t},s_{t + 1}})}\sim M},{a_{t + 1}\sim\pi_{\theta}}}\left\lbrack \text{⁠}{{\frac{1}{2}{Q_{\varphi_{i}}\left( {s_{t},a_{t}} \right)}} - \left( {{r\left( {s_{t},a_{t}} \right)} + {\gamma\left( {{Q_{\varphi^{\prime}}\left( {s_{t + 1},a_{t + 1}} \right)} - {{\alpha log\pi}_{\theta}\left( {a_{t + 1}❘s_{t + 1}} \right)}} \right)}} \right)^{2}} \right\rbrack}$

The weights φ′₁,φ′₂of the Target Soft-Q neutral networks are updated as follows:

φ′₁←φ+(1−τ)φ′₁

φ′₂←φ+(1−τ)φ′₂

A regularization coefficient α is updated, and its loss function is as follows:

J(α)=E[−αlog π_(t)(a _(t) |s _(t))−αH ₀]

step S56: determining whether step is greater than N, and if yes, proceeding to step S57; otherwise, incrementing step by 1,s_(t) ^(r)=s_(t+1) ^(r), s_(t) ^(b)=s_(t+1) ^(b), and skipping to step S53; and

step S57: determining whether the algorithm converges or whether training episodes are met, and if yes, ending the training and obtaining the trained SAC algorithm model; otherwise, skipping to step S51.

Further, the step S6 includes the following specific steps:

step S61: initializing the initial situations of both sides, and obtaining

s_(t) ^(r), s₁ ^(b) of the red and blue sides;

step S62: separately recording the states s_(t) ^(r), s_(t) ^(b), inputting the states s_(t) ^(r), s_(t) ^(b) to the Actor neutral network of the trained SAC algorithm model to output actions a_(t) ^(r), a_(t) ^(b) of the red and blue sides, and obtaining new states s_(t+1) ^(r), s_(t+1) ^(b) after performing the actions by both sides;

step S63: determining whether either of both sides succeeds in engaging in combat, and if yes, ending; otherwise, letting s_(t) ^(r)=s_(t+1) ^(r) and s_(t) ^(b)=s_(t+1) ^(b), and skipping to step S62;

step S64: plotting combat trajectories of both sides according to the recorded states s_(t) ^(r), s_(t) ^(b);

step S65: initializing the initial situations of n groups of UAVs on both sides, performing steps S62 to S63 on each group of UAVs on both sides, and finally recording whether either of both sides succeeds in engaging in combat, with the number of times of successfully engaging in combat being denoted as num; and

step S66: calculating num/n, namely a final combat success rate, to indicate the generalization capability of the decision-making model.

Specific Exemplary Embodiments

In the embodiment, when initializing a plurality of groups of UAVs on both sides, the combat area is x∈[−6 km,6 km],y∈[3km,4km],z∈[−6 km,6 km], and an initial velocity range is [50 m/s,400 m/s], while an initial pitch angle range is [−90°,90°] and an initial heading angle range is [−180°,180°].

The maximum attack range of a missile is 6 km and a minimum attack range is 1 km. The maximum off-boresight launch angle of the missile is 30°, w₁=w₂=0.5

The SAC algorithm model is constructed as follows: in the Actor neutral network in the SAC algorithm, the number of hidden layers l=2, and in each layer, the number of nodes n=256. The optimization algorithm is Adam algorithm, with discount factor γ=0.99, network learning rate lr=0.0003, entropy regularization coefficient α=1 and target entropy value H₀=−3

The number of parallel self-play environments is defined as env_num=[2,4,6,8,10,12,14,16,18,20]; the number of training sample groups is defined as batch_size=128; and the maximum simulation step size is defined as N=800.

After the training is finished, both sides are randomly initialized to test the trained algorithm, and combat trajectories are displayed, as shown in FIG. 3. As shown, after the combat is started, both red and blue sides make decisions using the trained model simultaneously and can effectively approach each other and attack the opponents.

200 Groups of UAVs on both sides are randomly initialized to test the trained algorithm, and a combat success rate is calculated. The results of the combat success rate varying with the number of parallel self-play environments is calculated as shown in FIG. 4. As shown, during training by parallel self-play, since the algorithm is characterized by sample sharing and strategy sharing, the model can comprehensively learn decisions suitable for all combat environments with increasing game environments, avoid overfitting, improve the generalization capability, allowing for optimal overall combat, and thus can reach a higher combat success rate in a new environment. When the number of the parallel game environments is 12, the decision model has the highest generalization capability and can achieve 69.15% combat success rate. With further increasing game environments, samples generated by the environments are too complicated, and the algorithm cannot allow agents in all environments to realize optimal combat and thereby will lose part of decision-making capability, leading to gradually reduced combat success rate.

Therefore, the maneuvering decision-making of UAVs can be effectively realized, and the generalization capability of the model can be improved, so that the model can be more practicable. 

What is claimed is:
 1. An air combat maneuvering method based on parallel self-play, comprising the following steps: step S1: constructing a unmanned aerial vehicle (UAV) maneuver model; step S2: defining our UAV as red side and enemy UAV as blue side; initializing both red and blue UAVs, and constructing a red-and-blue motion situation acquiring model to describe a relative combat situation of the red and blue sides; step S3: constructing state spaces S_(r),S_(b) of both red and blue sides, action spaces A_(r),A_(b) of both red and blue sides and a reward function R according to a Markov process; step S4: constructing a maneuvering decision-making model structure based on a soft actor-critic (SAC) algorithm; step S5: initializing a plurality of groups of UAVs on both sides, defining experimental parameters, and training the SAC algorithm by allowing the plurality of groups of UAVs on both sides to perform air combat confrontations using the same maneuvering decision-making model and a same replay buffer to realize parallel self-play; and step S6: randomly initializing both sides to test a trained network, and displaying combat trajectories; randomly initializing the plurality of groups of UAVs on both sides to test the trained network, and calculating a combat success rate.
 2. The air combat maneuvering method based on parallel self-play according to claim 1, wherein the constructing a UAV maneuver model specifically comprises the following steps: supposing an OXYZ coordinate system to be a three-dimensional spatial coordinate system for UAVs, wherein origin O represents the center of a combat area for UAVs, with X axis pointing to the north, Z axis pointing to the east and Y axis pointing in a vertical upward direction; regarding a UAV as a mass point and establishing equations of motion for the UAV as follows: $\left\{ \begin{matrix} X_{t + {dT}} & = & {X_{t} + {V_{t + {dT}}*{\cos\left( \theta_{t + {dT}} \right)}*{\cos\left( \varphi_{t + {dT}} \right)}*{dT}}} \\ Y_{t + {dT}} & = & {Y_{t} + {V_{t + {dT}}*{\sin\left( \theta_{t + {dT}} \right)}*{dT}}} \\ Z_{t + {dT}} & = & {Z_{t} + {V_{t + {dT}}*{{\cos\left( \theta_{t + {dT}} \right)}**{\sin\left( \varphi_{t + {dT}} \right)}}{dT}}} \end{matrix} \right.$ $\left\{ \begin{matrix} {V_{t + {dT}} = {V_{t} + {{dv}*{dT}}}} \\ {\theta_{t + {dT}} = {\theta_{t} + {d\theta*{dT}}}} \\ {\varphi_{t + {dT}} = {\varphi_{t} + {d\varphi*{dT}}}} \end{matrix} \right.$ wherein t denotes current time; dT denotes an integration step size of the UAV; [X_(t),Y_(t),Z_(t)], [X_(t+dT),Y_(t+dT),Z_(d+dT)] denote coordinate position components of the UAV at time t and time t+dT respectively; V_(t), V_(t+dT) denote velocities of the UAV at time t and time t+dT respectively; pitch angles θ_(t), θ_(t+dT) are included angles between velocity vectors of the UAV at time t and time t+dT, and XOZ plane; heading angles φ_(t), φ_(t+dT) are included angles between projection vectors of the velocity vectors of the UAV at time t and time t+dT on the XOZ plane, and the positive X axis; dυ denotes an acceleration of the UAV; dθ denotes a pitch angle variation of the UAV; and dφ denotes a heading angle variation of the UAV.
 3. The air combat maneuvering method based on parallel self-play according to claim 2, wherein the step S2 specifically comprises the following steps: describing the relative situation of both sides acquired by the red-and-blue motion situation acquiring model with {right arrow over (D)}, d and q, wherein {right arrow over (D)} denotes a position vector between the red side and the blue side in a direction from the red side to the blue side; d denotes a distance between the red side and the blue side; q denotes a relative azimuth angle, namely an included angle between the velocity vector {right arrow over (V)}_(r) and the distance vector {right arrow over (D)} of the red side; and denoting the combat situation of the blue side relative to the red side by {right arrow over (D)}_(r), d and q_(r) and the combat situation of the red side relative to the blue side by {right arrow over (D)}_(b), d and q_(b), wherein {right arrow over (D)}_(r) denotes a position vector between the red side and the blue side in a direction from the red side to the blue side; {right arrow over (D)}_(b) denotes a position vector between the blue side and the red side in a direction from the blue side to the red side; q_(r) denotes a relative azimuth angle of the blue side to the red side; and q_(b) denotes a relative azimuth angle of the red side to the blue side; and {right arrow over (D)}_(r), {right arrow over (D)}_(b), d, q_(r) and q_(b) are calculated as follows: ${\overset{\rightarrow}{D}}_{r} = \left( {{X_{b} - X_{r}},{Y_{b} - Y_{r}},{Z_{b} - Z_{r}}} \right)$ ${\overset{\rightarrow}{D}}_{b} = \left( {{X_{r} - X_{b}},{Y_{r} - Y_{b}},{Z_{r} - Z_{b}}} \right)$ $d = \sqrt{\left( {X_{b} - X_{r}} \right)^{2} + \left( {Y_{b^{-}}Y_{r}} \right)^{2} + \left( {Z_{b} - Z_{r}} \right)^{2}}$ $q_{r} = {{a{\cos\left( \frac{{\overset{\rightarrow}{D}}_{r} \times \overset{\rightarrow}{V_{r}}}{{\overset{\rightarrow}{D}}{\overset{\rightarrow}{V_{r}}}} \right)}} = {a{\cos\left( \frac{{\left( {X_{b} - X_{r}} \right)v_{xr}} + {\left( {Y_{b} - Y_{r}} \right)v_{yr}} + {\left( {Z_{b} - Z_{r}} \right)v_{zr}}}{d^{*}v_{r}} \right)}}}$ $q_{b} = {{{acos}\left( \frac{{\overset{\rightarrow}{D}}_{b} \times \overset{\rightarrow}{V_{b}}}{{\overset{\rightarrow}{D}}{\overset{\rightarrow}{V_{b}}}} \right)} = {a{\cos\left( \frac{{\left( {X_{r} - X_{b}} \right)v_{xb}} + {\left( {Y_{r} - Y_{b}} \right)v_{yb}} + {\left( {Z_{r} - Z_{b}} \right)v_{zb}}}{d^{*}v_{b}} \right)}}}$ wherein {right arrow over (R)}_(r)=(X_(r),Y_(r),Z_(r)), {right arrow over (V)}_(r)=(υ_(xr),υ_(yr),υ_(zr)), υ_(r), θ_(r) and φ_(r) are the position vector, velocity vector, velocity, pitch angle and heading angle of the red side, respectively; and {right arrow over (R)}_(b)=(X_(b),Y_(b),Z_(b)), {right arrow over (V)}_(b)=(υ_(xb),υ_(yb),υ_(zb)), υ_(b), θ_(b) and φ_(b) are the position vector, velocity vector, velocity, pitch angle and heading angle of the blue side, respectively.
 4. The air combat maneuvering method based on parallel self-play according to claim 3, wherein the step S3 specifically comprises the following steps: defining the state space of the red UAV as S_(r)=[X_(r),Y_(r),Z_(r),υ_(r),θ_(r),φ_(r), d,q_(r) ] and the state space of the blue UAV as S_(b)=[X_(b),Y_(b),Z_(b),υ_(b),θ_(b),φ_(b), d,q_(b) ]; defining the action space of the red UAV as A_(r)=[dυ_(r),dφ_(r),dθ_(r)] and the action space of the blue UAV as A_(b)=[dυ_(b),dφ_(b),dθ_(b)]; and forming the reward function R with a distance reward function R_(d) and an angle reward function R_(q), R=w₁*R_(d)+w₂*R_(a), wherein w₁,w₂ denote weights of a distance reward and an angle reward; the distance reward function R_(d) is expressed as: R_(d1) = −d/(5 * D_(max)) $R_{d2} = \left\{ \begin{matrix} {3,} & {{{if}D_{\min}} < d < D_{\max}} \\ {{- 1},} & {{{if}d} < D_{\min}} \end{matrix} \right.$ R_(d) = R_(d1) + R_(d2) wherein R_(d1) denotes a continuous distance reward while R_(d2) denotes a sparse distance reward; and D_(min) denotes a minimum attack range of a missile carried by the red side, while D_(max) denotes a maximum attack range of the missile carried by the red side; and the angle reward function R_(q) is expressed as: R _(q1) =−q/180 R _(q2)=3, if q<q _(max) R _(q) =R _(q1) +R _(q2) wherein R_(q1) denotes a continuous angle reward, while R_(q2) denotes a sparse angle reward; and q_(max) denotes a maximum off-boresight launch angle of the missile carried by the red side.
 5. The air combat maneuvering method based on parallel self-play according to claim 4, wherein the constructing a maneuvering decision-making model structure based on a SAC algorithm specifically comprises the following steps: generating maneuver control quantities for both red and blue sides by the maneuvering decision-making model based on the SAC algorithm using a SAC method, to allow the red and blue sides to maneuver; and implementing the SAC algorithm by neural networks comprising an replay buffer M, one Actor neural network π_(θ), two Soft-Q neural networks Q_(φ1) and Q_(φ2), two Target Soft-Q networks Q_(φ′) ₁ and Q_(φ′) ₂ , wherein θ, φ₁, φ₂, φ′₁, φ′₂ denote weights of the respective networks; wherein the Actor neural network π_(θ) receives an input of a state value s_(t) ^(r) of the red side or a state value s_(t) ^(b) of the blue side and generates outputs of mean μ(μ_(r),μ_(b)) and variance σ(σ_(r),σ_(b)); noise τ is generated by sampling from a standard normal distribution; an action a_(t) ^(r) of the red side or an action s_(t) ^(b) of the blue side is generated from the mean μ, variance σ and noise τ; the action a_(t) ^(r) or a_(t) ^(b) is limited to a range of (−1,1) by using a tanh function, and the process of generating the action is shown below: μ_(r),σ_(r)=π_(θ)(s_(t) ^(r)) μ_(b),σ_(b)=π_(θ)(s_(t) ^(b)) a _(t) ^(r) =N(μ_(r),σ_(r) ²)=μ_(r)+σ_(r)*τ a _(t) ^(b) =N(μ_(b),σ_(b) ²)=μ_(b)+σ_(b)*τ a_(t) ^(r)=tanh(a_(t) ^(r)) a_(t) ^(b)=tanh(a_(t) ^(b)) the Soft-Q neural networks Q₁₀₀ ₁ and Q_(φ2) receive inputs of a state value and an action value and output Q values predicted by the neural networks; the Target Soft-Q neural networks Q_(φ′) ₁ and Q_(φ′) ₂ have a same structure with and are different in network weight from the Soft-Q neural networks; the Soft-Q neural networks are configured to generate predicted Q values and select an action, while the Target Soft-Q neural networks are configured to generate target Q values and evaluate the action; the weights of the Soft-Q neural networks are updated in real time, and copied to the Target Soft-Q neural networks after multiple iterations; and each of the Actor, Soft-Q and Target Soft-Q networks is a fully-connected neutral network having l hidden layers, with n neurons in each hidden layer and an activation function ReLU.
 6. The air combat maneuvering method based on parallel self-play according to claim 5, wherein the step S5 specifically comprises the following steps: when initializing a plurality of groups of UAVs on both sides, with initial positions within the combat area, and setting an initial velocity range, an initial pitch angle range and an initial heading angle range; and the steps of training the SAC algorithm by performing air combat confrontations to realize parallel self-play are as follows: step S51: defining the number env_num of parallel self-play environments, defining the number batch size of batch training sample groups, defining a maximum simulation step size N, initializing step=1, initializing env=1, initializing initial situations of both sides, and obtaining an initial state s_(t) ^(r) of the red side and an initial state s_(t) ^(b) of the blue side; step S52: randomly generating Actor network weight θ, Soft-Q network weights φ₁, φ₂, initializing the policy network π_(θ) and the two Soft-Q networks Q_(φ1), Q₁₀₀ ₂, supposing φ′₁=φ₁, φ′₂=φ₂, and initializing the Target Soft-Q networks Q_(φ′) ₁ , Q_(φ′) ₂ with φ′₁, φ′₂ as network weights; step S53: inputting a state s_(t) ^(r) of the red side to the Actor network to output a mean μ_(r) and a variance σ_(r), obtaining an action a_(t) ^(r) that fits the action space A_(r) in step S3 from the process of generating the action in step S4, obtaining a new state by the red side after performing the action s_(t+1) ^(r), and obtaining a reward value r_(t) ^(r) according to the reward function R in step S3; inputting a state s_(t) ^(b) of the blue side to the Actor network to output a mean μ_(b) and a variance σ_(b), obtaining an action a_(t) ^(b) that fits the action space A_(b) in step S3 from the process of generating the action in step S4, obtaining a new state by the blue side after performing the action s_(t+1) ^(b), and obtaining a reward value r_(t) ^(b) according to the reward function R in step S3; and storing tuple <s_(t) ^(r),a_(t) ^(r),s_(t+1) ^(r),r_(t) ^(r)> and tuple <s_(t) ^(b),a_(t) ^(b),s_(t+1) ^(b),r_(t) ^(b)> in the replay buffer M; step S54: determining whether env is greater than env_num, and if yes, proceeding to step S55; otherwise, incrementing env by 1, and skipping to step S51; step S55: when the number of experience groups in the replay buffer is greater than batch_size, randomly sampling batch_size groups of experience to update parameters of the Actor and Soft-Q neutral networks in the SAC algorithm, and updating a regularization coefficient α; step S56: determining whether step is greater than N, and if yes, proceeding to step S57; otherwise, incrementing step by 1 s_(t) ^(r)=s_(t+1) ^(r), s_(t) ^(b)=s_(t+1) ^(b), and skipping to step S53; and step S57: determining whether the algorithm converges or whether training episodes are met, and if yes, ending the training and obtaining the trained SAC algorithm model; otherwise, skipping to step S51.
 7. The air combat maneuvering method based on parallel self-play according to claim 6, wherein the step S6 specifically comprises the following steps: step S61: initializing the initial situations of both sides, and obtaining the initial states s_(t) ^(r), s_(t) ^(b) of the red and blue sides; step S62: separately recording the states s_(t) ^(r), s_(t) ^(b), inputting the states s_(t) ^(r), s_(t) ^(b) to the Actor neutral network of the trained SAC algorithm model to output actions a_(t) ^(r), a_(t) ^(b) of the red and blue sides, and obtaining new states s_(t+1) ^(r), s_(t+1) ^(b) after performing the actions by both sides; step S63: determining whether either of both sides succeeds in engaging in combat, and if yes, ending; otherwise, supposing s_(t) ^(r)=s_(t+1) ^(r) and s_(t) ^(b)=s_(t+1) ^(b), and skipping to step S62; step S64: plotting combat trajectories of both sides according to the recorded states s_(t) ^(r), s_(t) ^(b); step S65: initializing the initial situations of n groups of UAVs on both sides, performing steps S62 to S63 on each group of UAVs on both sides, and finally recording whether either of both sides succeeds in engaging in combat, with the number of times of successfully engaging in combat being denoted as num; and step S66: calculating num/n, namely a final combat success rate, to indicate the generalization capability of the decision-making model.
 8. The air combat maneuvering method based on parallel self-play according to claim 7, wherein in the step S5, the initial velocity range is set as [50 m/s,400 m/s], and the initial pitch angle range as [−90°,90°] and the initial heading angle range as [−180°,180°]. 